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ABSTRACT 



A concept map is a graphical representation of a person's 
(student's) knowledge of a domain. Concept maps have been used in educational 
settings for some time and many computer-based implementations of interactive 
concept map building tools exist. These concept mapping tools often provide 
for a solely prepositional, primarily textual, knowledge representation 
scheme and do not fully capitalize on the functionality offered by the 
computational medium. This paper offers psychological and pedagogical design 
rationales for inclusion of multimedia in computer-based concept maps. It 
also describes a concept mapping application named Webster whose goals 
include more comprehensively representing students' knowledge of a domain, 
providing facilities that make concept maps more pedagogically effective for 
students using them to learn new concepts, and in doing both, capitalizing 
more fully on the capabilities of the computational medium. There are a 
number of characteristics of Webster that attempt to achieve these desiderata 
but this paper focuses in particular on Webster's use of multimedia to 
realize these goals. (Contains 19 references.) (Author/AEF) 
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Abstract: A concept map is a graphical representation of a person’s (student’s) knowledge of a domain. Concept maps 
have been used in educational settings for some time and many computer-based implementations of interactive concept 
map building tools exist. These concept mapping tools often provide for a solely propositional, primarily textual, 
knowledge representation scheme and do not fully capitalize on the functionality offered by the computational 
medium. In this paper, we motivate the enhancement of “traditional” computer-based concept mapping tools with 
dynamic multimedia — sound, video, and animated images — which offer representational capabilities that are 
analogous to human knowledge representation mechanisms. 

Concept maps 

A concept map ™ is a graphical representation of knowledge of a domain. A concept map consists of nodes 
representing concepts, objects, or actions, connected by directional links defining the relationships between nodes. A node is 
represented by a simple geometric object, such as an oval, containing a textual concept name. Inter-node relationship links are 
represented by textually labeled lines with an arrowhead at one or both ends. Together, nodes and links define propositions , 
assertions about the domain. For example, the nodes and link in Figure 1 represent the proposition “horses can gallop” and 
might be a portion of a concept map about horses. 



Figure 1 . rses can gallop.” 

Concept maps have been used in the educational community for decades, in virtually every subject area. In 



educational : 



eir knowledge 



of a domain, may be used for teacher assessment of students’ knowledge of a topic covered in class, act as tools to aid study 
and comprehension of a domain or story, support idea generation and organization in preparation for prose composition, and 
are used as instructional materials for learning new concepts and their interrelationships. 

This paper offers psychological and pedagogical design rationales for the inclusion of multimedia in computer- 
based concept maps. It also describes a concept mapping application named Webster 1 ’ 1 whose goals include more 
comprehensively representing students’ knowledge of a domain, providing facilities that make concept maps more 
pedagogically effective for students using them to learn new concepts, and in doing both, capitalizing more fully on the 
capabilities of the computational medium. There are a number of characteristics of Webster that attempt to achieve these 
desiderata but in this paper we focus in particular on Webster’s use of multimedia to realize these goals. 



Why Multimedia Concept Maps? 

The purpose of concept maps is to visually represent knowledge of a subject or domain. Indeed, Jonassen (1992) 
claims that concept maps are accurate reflections of their authors’ cognitive structures. But this argument is based on concept 
maps that portray text-based propositions only. Jonassen goes on to pose the question “What constraints does the software 
impose on the product?” (p. 20). Older concept map tools are indeed very good at representing text-based propositional 
statements — wherein concepts are described by verbal means alone — but not necessarily other forms of knowledge. However, 
if concept maps are to be used to externally represent one’s (internal) knowledge, they should allow for nodes to be 
something other than of a verbal or textual nature. “It is important to note that not all nodes in a semantic memory system 
have names corresponding to words in natural language” (Rumelhart & Norman, 1985, p. 24). This notion is further 
supported by numerous researchers who have offered evidence or arguments for the notion that one’s knowledge contains 
more than simply verbal propositional knowledge. Kosslyn (1980), for example, proposes a cognitive model that represents 
information about objects with both propositional properties and images. Johnson-Laird (1983) similarly asserts that mental 
models of a domain include both propositions and imagery. Baddeley adds there is “abundant evidence for separate visual 
and verbal coding” (1985, p. 212). Shephard (1967) has, furthermore, provided evidence that memory for visual imagery is 
more robust than that for purely textual information. And Paivio (1986) asserts information encoded both visually and 
verbally is more memorable. So, to begin with, we see a need for image-based nodes in concept maps if we wish to more 
comprehensively represent one’s knowledge of a domain or use maps to convey information new to learners. 
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Intuitively, in addition to static imagery, temporally dynamic visual and auditory memories are also part of one's 
knowledge of an object or domain. Here we’re speaking of images in motion and sounds, both of which occur over time. 
Indeed, this notion has been acknowledged by cognitive science researchers and others. Johnson-Laird (1983) asserts that 
some mental representations are of temporally dynamic nature. Numerous studies involving music cognition provide strong 
empirical evidence for long-term auditory memory (see, e.g., Dowling & Harwood, 1 986). Even popular culture reflects these 
notions; for example, as Lennon and McCartney wrote and the Beatles sang, “Penny Lane is in my ears and in my eyes,” 
highlighting the reality that recollections involve long-term auditory and dynamic visual memories. Hence, one’s knowledge 
of horses might include not only what a horse looks like, but what a horse looks like when galloping and when trotting, the 
image of a jumping horse gliding over an obstacle, the sight and thundering sound of a group of wild horses stampeding 
across a sandy plain, the moving images of a wild bronco bucking in its attempt to throw a rodeo cowboy off its back, the 
sound of a horse “neighing.” Certainly these are parts of our long-term memories for horses, and we ought to be able to 
represent these memories in concept maps to demonstrate our own knowledge or to use concept maps as instructional 
resources. 

Imagery in general also offers the important capability of reifying concepts described by text. The ability to 
represent concrete instances and link them to text-based concepts supports the learning and memorability of those concepts. 
For example, Moore and Skinner (1 985) demonstrated that abstract textual information is better understood and learned when 
presented along with reifying illustrations. The ability to portray concrete instances of concepts adds significantly to the 
representational and instructional potential of maps. The ability to use dynamic imagery and audio significantly enhances this 
representational capability. For example, here is a definition of the trotting gait of a horse: “In this most diagonal gait, the 
diagonal hooves lift off from and hit the ground at the same moment. [Right hind, left front, alternate with left hind, right 
front.] To see this, focus on a front foot, then include the diagonal hind in your field of vision. You can clearly see them lift 
off and hit together, in a hard trot. The sound will be a 1-2 beat” (Ziegler, 1988). Compare a concept map that represents 
these ideas via text-only nodes to another map that accompanies this information with a slow-motion video of a horse actually 
trotting. Clearly, the latter not only makes the conceptual information concrete but also provides an additional dimension to 
the map, namely sense-based information the text alone cannot. 

The incorporation of temporally dynamic visual and aural elements in a concept map knowledge representation tool 
enhances its “flexibility of expressiveness” (Heeren & Kommers, 1 992). More semantic and aesthetic options are open for the 
user to decide upon. This may help to accommodate individual differences, that is, to better support students with different 
expressive or learning preferences or needs when using concept maps to demonstrate their own knowledge or acquire new 
knowledge. And, clearly, temporally dynamic content in concept maps will better support those using maps for learning new 
domains and concepts because these users will be presented with: what objects look like when in motion, what they sound 
like, how things move or sound in particular contexts or situations, instructional visualizations of how to perform procedural 
tasks, videos showing dynamic interactions between objects or people. 

Lastly, the use of multimedia in an interactive concept map builder and viewer may provide for a more engaging 
user experience. Of course, engaged and motivated learners are a desirable effect of any educational environment or tool 
(Vroom, 1994). 

If we are using the computer to represent students’ knowledge, we should more fully benefit from what the 
computational medium offers over more static media, such as pencil and paper. This includes the ability to incorporate 
dynamic content in concept maps, thereby more comprehensively representing the knowledge a person possesses. If we are 
asking students to represent their knowledge, we need to be cognizant of the fact that some aspects of a student’s mental 
model may be expressible only through visual or auditory media. And for students using concept maps to learn about new 
concepts, the enhancement of those maps with dynamic media has the potential to improve the maps’ pedagogical 
effectiveness. Of course, one’s knowledge of a domain may also contain tactile, olfactory, or taste memories. However, 
standard PCs are not currently capable of representing and “playing back” such sensory information, notwithstanding ongoing 
research in these areas. But they are capable of manipulating video, audio, and animation. 

To recap, concept map tools should be able to represent multiple types or forms of knowledge, not merely text- 
based propositions. Specifically, incorporating multimedia in concept mapping software should (a) provide for greater 
cognitive fidelity in student-constructed concept maps, allowing students to more comprehensively represent their knowledge 
in ways similar to their own cognitive representations; (b) offer the illustrative advantages of dynamic visual imagery and 
audio to students learning new concepts and domains; (c) provide the capability of reifying concepts with concrete instances 
that can be seen and heard; (d) offer richer expressive power for concept map authors; (e) provide for a more engaging 
student experience; and (0 better capitalize on functionality available in modem personal computers. 
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Figure 2. A Webster concept map, in a Web browser window, with a static image (the horseshoes), an animated 
image node (the galloping horse), an audio element (represented by the radio node), three video nodes (the 
television nodes), and a hyperlink to an external Web site (equestria.net). 



Multimedia in Webster 

On the basis of the preceding cognitive and pedagogical design rationale, Webster allows for the representation of 
dynamic media components as integral constructs of concept maps. That is, Webster provides tools for the facile 
incorporation of animated images, video, and audio in concept maps. In Webster, multimedia elements are first-class 
elements of concept maps with the same status and ease-of-use as textual concept nodes. 

Figure 2 shows Webster running in a Web browser. Webster is implemented in Java™ and thereby runs as an 
applet in standard Web navigators, providing student access from anywhere they have access to the Internet. Along the left 
side of Webster’s user interface are tools for creating elements that may reside in a Webster concept map: concept nodes (of 
various shapes; the user may assign different semantics to different shapes as appropriate for each map), relationship links, 
image, audio, and video nodes, and other types of map elements. At the top of this toolkit are two mutually exclusive toggle 
buttons (“iconic radio buttons”) that determine whether the user is in author mode (the pencil icon) or viewer mode (the eye). 
A user can create or modify maps while in author mode; in viewer mode, the map’s knowledge elements may not be edited 
and it is in this mode that multimedia elements may be viewed. When a teacher reviews a student’s concept map, he or she 
has access to viewer mode only for that map. 
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Figure 3. When a user clicks on a 
television node while in viewer mode, 
Webster directs its Web browser to 
download and play the node’s 
associated video file. Radio/audio 
nodes are handled in the same fashion. 



Continuing with our example domain, suppose a student is building a concept map to demonstrate her knowledge of 
horses. In addition to propositional assertions such as “a horse is an animal,” “a horse can gallop,” and “domesticated horses 
wear horseshoes” the student wishes to represent what a typical horse sounds like, what it looks like when galloping, and so 
on. She therefore requires the ability to incorporate visual imagery and audio in her map. To incorporate sound, she includes 
an audio node in the map; similarly for video and image data and their respective nodes. To add an audio node to a concept 
map in Webster, the user clicks on the “radio” tool button in the toolkit on the left and drops a radio (that is, audio) element 
onto the map. This is the same process used to add textual concept nodes (and other node types) to a map; multimedia nodes 
are not handled in any sort of special case fashion. Webster responds to the addition of the new audio node by popping up a 
list of available audio files (only audio files are shown for a radio node), allowing the user to select one to associate with the 
node. Image and video nodes are handled in the identical manner using the “portrait” tool and “television” tools, respectively. 
The multimedia file associated with an existing media node may be modified by double-clicking on the particular video, 
audio, or image node; Webster again displays a list of files of the appropriate media type for the user to select from. This is 
another example of how multimedia knowledge elements do not involve special case handling; double-clicking to edit a 
node’s content is precisely how textual concept and Web-link nodes are modified. 

To listen to an audio, the user (say, a teacher viewing a completed concept map) simply clicks on a radio node in 
the map (while in viewer mode). In response, Webster plays the associated audio file. Video playback is similarly invoked via 
TV nodes in a map (see Figure 3). 

With regard to images, there are other concept map tools that allow the use of static images as nodes in a map (see 
Related Work). Webster goes a step further in that image-based nodes in a concept map may be animated-GIF images. Thus 
we may have animations directly within a concept map. For example, the animated horse image in Figure 2 appears to be 
galloping within the map. Image nodes may be sized (stretched) directly within the map by the map author to show or hide 
details or otherwise accommodate aesthetic and spatial considerations. If a user wishes to view an image in its originally- 
created size, she may click on the image node (while in viewer mode) and the full-sized image is displayed in the browser 
frame that exists below the Webster applet (not shown in Figure 2). 

Webster also automates the conversion of concept maps to a different representation form, namely outlines. A 
common activity associated with concept map usage in the classroom is generating and organizing ideas in preparation for 
writing a report, composition, story, or, a more modem form of “composition,” a Web page. Of course, a common 
representation for idea organization is an outline. Further, alternative representations support developing deeper 
understanding of a domain. Hence, as in the Inspiration® concept mapping product (more on Inspiration below), Webster can 
translate a map into an outline containing the map’s semantic content. 

In Webster, multimedia concept maps become multimedia outlines. The same static and animated images, videos, 
and audios that are part of a concept map appear in the associated outline representation, as shown in Figure 4. Webster 
converts maps into outlines as HTML documents that appear in a separate Web browser window. As in the concept map 
itself, the HTML generated by Webster embeds radio images for audio nodes and a TV for each video. Outlines incorporate 
the text labels that appear on relationship links in the map, so an outline item (a single line in the outline) might appear with 
the text “looks like:” followed by a TV image. In the HTML document, the radio and TV images are coded as hyperlinks that 
reference specific audio or video files. Hence, users simply click on embedded radio and television images to “play” them. 
With regard to static and animated images, they are handled much as they are in concept maps: images are embedded and 
visible directly within outline items, so, for example, once again the horse seems to be galloping within the outline. All 
images are displayed in the outline with the same default thumbnail dimensions. These thumbnail images are also encoded in 
the HTML as hyperlinks to the full-size image file, so to view the image in its original size, a user simply clicks on the 
thumbnail in the outline. 
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Figure 4. The concept map 
shown in Figure 2 
translated to a multimedia 
outline displayed as a 
Web/HTML page. The 
animated horse appears to 
be galloping, the radio and 
TV images are hyperlinks 
to audio and video files. 



Related Work 

There are many existing computer-based concept map tools, far too many for exhaustive description here. But, 
some bear mention. With regard to images, Inspiration and Axon®, two commercial concept mapping products, and 
SemNet™, a shareware tool (Fisher, 1992), do allow the use of static images as nodes in a map. Another mapping tool 
permits the association of static imagery with a concept node: Learning Tool™ (Kozma, 1992) has the provision for a 
“notecard” to be associated with each concept node, and this notecard may contain text and images. These tools certainly 
offer a step in the right direction beyond text-only representations. But we should be able to incorporate temporally dynamic 
content in concept maps as well. Further, there are significant differences in the outlines generated by Inspiration and Webster 
(we’ll focus only on the multimedia aspect). Even though Inspiration incorporates only static image and textual nodes in 
concept maps, the images are absent in its outline translation of a map, whereas Webster’s outlines incorporate all the media 
elements that appear in the corresponding concept map. None of these mapping tools is accessible from a Web browser. 

HandLeR is a handheld computer running proprietary software which includes a conceptual mapping tool that 
incorporates multimedia elements (Sharpies, 2000a). In HandLeR’s “topic map” tool, topic nodes may reference the same 
types of media files as supported in Webster (Sharpies, 2000b). There are a number of differences between HandLeR’s map 
tool and Webster. Topic nodes in HandLeR are all identical ovals (with embedded text), connected by unlabeled links. Hence, 
static and animated images are not visible within the map itself. The further implication is that nodes are not “typed” — this is, 
different node types are not visually distinguished — whereas in Webster concept maps, nodes provide visual cues regarding 
their content, including the clear indication of the existence of media nodes (video nodes appear as “televisions” and audio 
nodes as “radios”). We refer to this as visually typing , that is, making each node’s content type evident by its appearance. 
Unlike HandLeR, Webster provides an alternative outline representation that incorporates multimedia as well. Lastly, 
Webster is Web-enabled, providing broader student access to the tool in terms of location and time as well as eliminating the 
logistical problems of distributing software to individuals, while HandLeR operates on a standalone basis. 

In the IHMC CMap toolkit (IHMC, 2000) users may associate resources with a concept node. Resources may 
include images, sounds, or video files. Again, images do not appear directly within the visible concept map per se — users 
must follow a hyperlink to view an image, and this tool does not incorporate outline translations (multimedia-based or not) of 
concept maps. There are no labeled relationship links between verbal concepts and media nodes; media resources are, in a 
sense, part of the concept nodes. We feel that labeled links showing the relationships between media nodes and other nodes 
represent additional knowledge thereby enhancing the semantics of a map, (Link labels can be omitted in Webster if desired; 
making labels optional and the possible enhanced semantics when links are included increase Webster’s flexibility of 
expressiveness for knowledge representation — more options and decisions are left to the user’s preferences and 
representational requirements.) Indeed, based on studies with learning disabled students, Griffin, Simmons, and Kameenui 
(1992) strongly recommend that concept maps and other graphical organizers explicitly depict (i.e., label) the relationships 
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among conceptual elements to better support the learning and memorability of the represented domain. Further, Webster 
attempts to make the representation of laiowledge elements straightforward and simple, both conceptually and in terms of 
user interaction, in part by treating different types of knowledge elements in a uniform fashion. Multimedia knowledge is 
represented as distinct nodes which are manipulated in the same manner as are other nodes in a map. Plus, in our design 
rationale we stated that there are apparently distinct cognitive systems for the representation of verbal and imaginal elements; 
if we wish to reflect one’s knowledge in a concept map, it seems to follow that knowledge elements of these types be 
represented by distinct nodes rather than bundled into a single node. 

Conclusion 

A concept map is a tool for representing knowledge. In asking students to construct concept maps, we are asking 
them to demonstrate and communicate to others their knowledge of a domain. We ought not limit those students by restricting 
the types of knowledge they can they can represent. Webster offers greater expressiveness and broader representation 
capabilities by incorporating multimedia elements in concept maps, thereby attempting to offer a closer match to cognitive 
knowledge representations. The incorporation of dynamic media also provides concomitant benefits to students using pre- 
constructed concept maps as tools for new concept learning, because they are exposed to a broader range of knowledge 
media. Lastly, a concept mapping tool should be able to translate maps into alternative representations, such as outlines which 
many students find useful or necessary for composing narratives. Webster outlines contain all of a map’s content including its 
multimedia knowledge elements. 
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